<html>
<head><meta charset="utf-8"><title>meeting 2021-05-31 · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html">meeting 2021-05-31</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="240853579"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240853579" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240853579">(May 31 2021 at 16:52)</a>:</h4>
<p><span class="user-group-mention" data-user-group-id="2943">@T-infra</span> T-7min</p>



<a name="240853621"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240853621" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240853621">(May 31 2021 at 16:53)</a>:</h4>
<p>I’m traveling today so mostly read-only. (Holiday in the US today)</p>



<a name="240853841"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240853841" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240853841">(May 31 2021 at 16:56)</a>:</h4>
<p>huh TIL may 31 is holiday in the US <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="240854331"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854331" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854331">(May 31 2021 at 17:02)</a>:</h4>
<p><span class="user-group-mention" data-user-group-id="2943">@T-infra</span> meeting time!</p>



<a name="240854341"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854341" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854341">(May 31 2021 at 17:02)</a>:</h4>
<p>o/</p>



<a name="240854346"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854346" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854346">(May 31 2021 at 17:02)</a>:</h4>
<p>o/</p>



<a name="240854361"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854361" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854361">(May 31 2021 at 17:02)</a>:</h4>
<p>huh, also happens to be holiday in uk</p>



<a name="240854406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854406">(May 31 2021 at 17:03)</a>:</h4>
<p>am I the only one <em>not</em> on holiday here? <span aria-label="laughing" class="emoji emoji-1f606" role="img" title="laughing">:laughing:</span></p>



<a name="240854503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854503">(May 31 2021 at 17:04)</a>:</h4>
<p>ok let's start</p>



<a name="240854521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854521">(May 31 2021 at 17:04)</a>:</h4>
<p>we don't have any issues to look at, and the only item on the agenda is <a href="http://crates.io">crates.io</a> monitoring</p>



<a name="240854542"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854542" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854542">(May 31 2021 at 17:04)</a>:</h4>
<p>I actually have news!</p>



<a name="240854684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854684">(May 31 2021 at 17:06)</a>:</h4>
<p>this morning I looked if there were any out-of-the-box solutions that would allow us to both collect heroku postgresql metrics and instance-level <a href="http://crates.io">crates.io</a> metrics, and I stumbled upon <a href="https://vector.dev/">https://vector.dev/</a></p>



<a name="240854706"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854706" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854706">(May 31 2021 at 17:06)</a>:</h4>
<p>oh oops, I've been using that personally for a while <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="240854708"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854708" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854708">(May 31 2021 at 17:06)</a>:</h4>
<p>Vector is a tool (written in Rust <span aria-label="tada" class="emoji emoji-1f389" role="img" title="tada">:tada:</span>) that allows to receive, transform and forward <em>both</em> metrics and logs</p>



<a name="240854735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854735">(May 31 2021 at 17:07)</a>:</h4>
<p>and it has support for custom lua scripts doing transformations</p>



<a name="240854789"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854789" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854789">(May 31 2021 at 17:07)</a>:</h4>
<p>I spent this afternoon playing around with it, and I now have a proof-of-concept docker container that receives the postgres log messages and exports them as prometheus metrics</p>



<a name="240854854"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854854" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854854">(May 31 2021 at 17:08)</a>:</h4>
<p>oh wait - so it can transform logs <em>into</em> metrics?</p>



<a name="240854866"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854866" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854866">(May 31 2021 at 17:08)</a>:</h4>
<p>with custom lua scripts yes</p>



<a name="240854868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854868">(May 31 2021 at 17:08)</a>:</h4>
<p>that I did not realise</p>



<a name="240854874"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854874" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854874">(May 31 2021 at 17:08)</a>:</h4>
<p><a href="https://github.com/pietroalbini/crates-io-heroku-metrics/blob/7bb68389cc61d817817e45f8aef00a003c05a702/src/vector.toml#L34-L71">https://github.com/pietroalbini/crates-io-heroku-metrics/blob/7bb68389cc61d817817e45f8aef00a003c05a702/src/vector.toml#L34-L71</a></p>



<a name="240854879"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854879" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854879">(May 31 2021 at 17:08)</a>:</h4>
<p>took me a while to realize it too after looking how to do it in the docs</p>



<a name="240854887"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854887" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854887">(May 31 2021 at 17:08)</a>:</h4>
<p>nifty</p>



<a name="240854921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854921">(May 31 2021 at 17:09)</a>:</h4>
<p>looks a bit hacky, but not too bad</p>



<a name="240854928"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240854928" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240854928">(May 31 2021 at 17:09)</a>:</h4>
<p>it's a bit hacky but it relies on fully documented behavior</p>



<a name="240855047"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855047" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855047">(May 31 2021 at 17:11)</a>:</h4>
<p>yeah seems pretty good</p>



<a name="240855076"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855076" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855076">(May 31 2021 at 17:11)</a>:</h4>
<p>sounds brill</p>



<a name="240855084"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855084" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855084">(May 31 2021 at 17:11)</a>:</h4>
<p>so, my current thinking would be to use vector and:</p>
<ul>
<li>for heroku postgres use a custom lua transform to parse the log line and turn it into multiple metrics</li>
<li>for instance-level, have the application output the datastructures Vector expects into a single log line, and have a custom lua transform to split them into multiple events/metrics</li>
</ul>



<a name="240855227"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855227" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855227">(May 31 2021 at 17:13)</a>:</h4>
<p>so this would be reimplementing some of the instance-level datagathering? that would clasically be done by <code>node_exporter</code></p>



<a name="240855236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855236">(May 31 2021 at 17:13)</a>:</h4>
<p>Does that start to give us anything towards an actual logging platform?</p>



<a name="240855255"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855255" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855255">(May 31 2021 at 17:13)</a>:</h4>
<p>in my proof of concept I also have nginx in the container, adding authentication and merging the multiple ports into a single one (the heroku source and the prometheus sink otherwise have different ports)</p>



<a name="240855335"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855335" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855335">(May 31 2021 at 17:14)</a>:</h4>
<p><span class="user-mention silent" data-user-id="117568">Aidan Hobson Sayers</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-05-31/near/240855227">said</a>:</p>
<blockquote>
<p>so this would be reimplementing some of the instance-level datagathering? that would clasically be done by <code>node_exporter</code></p>
</blockquote>
<p>during this thing for instance-level I meant instance-level <em>application</em> metrics, like request counts, in-flight requests and database pool status</p>



<a name="240855384"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855384" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855384">(May 31 2021 at 17:15)</a>:</h4>
<p>this doesn't cover <code>node_exporter</code>-like metrics on the dyno unfortunately, worst case scenario we could have <code>vector</code> on the dynos too writing to our main <code>vector</code> instance</p>



<a name="240855449"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855449" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855449">(May 31 2021 at 17:16)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116155">Jake Goulding</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-05-31/near/240855236">said</a>:</p>
<blockquote>
<p>Does that start to give us anything towards an actual logging platform?</p>
</blockquote>
<p>hmm, we're already using cloudwatch logs for stuff on aws and papertrail for heroku, is there something you'd need on the playground?</p>



<a name="240855587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855587">(May 31 2021 at 17:18)</a>:</h4>
<p>so, is there any concern on this approach or should we go with it?</p>



<a name="240855633"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855633" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855633">(May 31 2021 at 17:19)</a>:</h4>
<p>I'm still a bit unhappy but it doesn't sound like there's not <em>really</em> significantly better options</p>



<a name="240855650"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855650" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855650">(May 31 2021 at 17:19)</a>:</h4>
<p>When doing Prometheus stuff, it was recommended to avoid having “too much” in Prometheus and to try and do more via logs. I didn’t think we had a way to aggregate logs and then surface pretty graphs from them</p>



<a name="240855661"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855661" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855661">(May 31 2021 at 17:19)</a>:</h4>
<p>I think writing up the options we considered -- similar to the doc you had already prepared for the remote write option -- and posting a bit to internals or so to see if there's any strongly negative reactions or so</p>



<a name="240855778"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855778" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855778">(May 31 2021 at 17:21)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-05-31/near/240855633">said</a>:</p>
<blockquote>
<p>I'm still a bit unhappy but it doesn't sound like there's not <em>really</em> significantly better options</p>
</blockquote>
<p>I'm wondering what improvement would you like to see</p>



<a name="240855807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855807">(May 31 2021 at 17:21)</a>:</h4>
<p>oh, mostly just feels unfortunate to have so many moving pieces and services</p>



<a name="240855852"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855852" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855852">(May 31 2021 at 17:22)</a>:</h4>
<p>but I'm not actually seeing ways to avoid it really</p>



<a name="240855872"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855872" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855872">(May 31 2021 at 17:22)</a>:</h4>
<p>just feels like there's something wrong underlying it all, so to speak</p>



<a name="240855892"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855892" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855892">(May 31 2021 at 17:22)</a>:</h4>
<p>heh, mainly it's that heroku doesn't expose the metrics in any way other than log messages :(</p>



<a name="240855915"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855915" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855915">(May 31 2021 at 17:23)</a>:</h4>
<p>IMO that's the underlying thing that's wrong</p>



<a name="240855919"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855919" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855919">(May 31 2021 at 17:23)</a>:</h4>
<p>well, sure, though that's not the full scope, but yeah -- I mean ultimately I'm not against this, just unhappy :)</p>



<a name="240855952"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855952" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855952">(May 31 2021 at 17:23)</a>:</h4>
<p>the eternal cry when working with computers - "there must be a better way" <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="240855961"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240855961" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240855961">(May 31 2021 at 17:23)</a>:</h4>
<p>certainly it doesn't feel elegant</p>



<a name="240856007"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856007" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856007">(May 31 2021 at 17:24)</a>:</h4>
<p>but oh well</p>



<a name="240856064"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856064" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856064">(May 31 2021 at 17:24)</a>:</h4>
<p>I do think the internals post is not a bad idea, as it lets us source from a lot more knowledge/history</p>



<a name="240856100"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856100" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856100">(May 31 2021 at 17:25)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-05-31/near/240855661">said</a>:</p>
<blockquote>
<p>I think writing up the options we considered -- similar to the doc you had already prepared for the remote write option -- and posting a bit to internals or so to see if there's any strongly negative reactions or so</p>
</blockquote>
<p>hmm, how useful do you think this would be? I'd really like to have metrics up soon and I'm not sure whether there are a lot of people with experience in our exact stack on internals</p>



<a name="240856178"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856178" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856178">(May 31 2021 at 17:26)</a>:</h4>
<p>I mean our exact stack is ... heroku and prometheus, right?</p>



<a name="240856224"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856224" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856224">(May 31 2021 at 17:26)</a>:</h4>
<p>that seems 'hopefully common'</p>



<a name="240856232"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856232" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856232">(May 31 2021 at 17:26)</a>:</h4>
<p>I don't think it needs to block metrics, to be clear</p>



<a name="240856239"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856239" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856239">(May 31 2021 at 17:26)</a>:</h4>
<p>well, due to the limitations I'm not sure how common it is <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="240856261"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856261" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856261">(May 31 2021 at 17:26)</a>:</h4>
<p>but I think having it up even as a documentation piece that we can refer back to when looking back would be good</p>



<a name="240856284"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856284" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856284">(May 31 2021 at 17:27)</a>:</h4>
<p>and, hey, if someone proposes a much better option we can always put migrating to it on the issue log</p>



<a name="240856328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856328">(May 31 2021 at 17:27)</a>:</h4>
<p>one option is implementation to try it out and shake out any issues, then documenting with architecture diagrams/whatever and posting on internals for feedback</p>



<a name="240856385"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856385" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856385">(May 31 2021 at 17:28)</a>:</h4>
<p>yeah</p>



<a name="240856393"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856393" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856393">(May 31 2021 at 17:28)</a>:</h4>
<p>sure, that seems reasonable, order doesn't matter as much to me</p>



<a name="240856394"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856394" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856394">(May 31 2021 at 17:28)</a>:</h4>
<p>ok, so, I'll move the proof of concept repo to the rust-lang org, polish it up and deploy it to Fargate in the coming days</p>



<a name="240856405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856405">(May 31 2021 at 17:28)</a>:</h4>
<p>I just want some documentation-y stuff, and posting that just seems like an easy possible (though unlikely) win</p>



<a name="240856421"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856421" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856421">(May 31 2021 at 17:29)</a>:</h4>
<p>I'll definitely write docs on its design</p>



<a name="240856435"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856435" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856435">(May 31 2021 at 17:29)</a>:</h4>
<p>ok! I have a really quick thing to discuss in private, otherwise thanks everyone!</p>



<a name="240856436"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856436" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856436">(May 31 2021 at 17:29)</a>:</h4>
<p>\o</p>



<a name="240856461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856461">(May 31 2021 at 17:29)</a>:</h4>
<p>\o</p>



<a name="240856855"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240856855" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240856855">(May 31 2021 at 17:34)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116155">Jake Goulding</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-05-31/near/240855650">said</a>:</p>
<blockquote>
<p>When doing Prometheus stuff, it was recommended to avoid having “too much” in Prometheus and to try and do more via logs. I didn’t think we had a way to aggregate logs and then surface pretty graphs from them</p>
</blockquote>
<p>hmm, we don't have that right now -- we're at a scale where moving every metric and graph to prometheus should work just fine, unless it's something hard to gather with prometheus</p>



<a name="240857069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240857069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240857069">(May 31 2021 at 17:37)</a>:</h4>
<p>I also am not sure how much that recommendation really makes sense -- it's my understanding that there's basically just two separate <em>kinds</em> of data, some is best sourced through logs, some is best sourced through metrics -- basically unbounded value sets vs. bounded value sets</p>



<a name="240857081"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-05-31/near/240857081" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-05-31.html#240857081">(May 31 2021 at 17:37)</a>:</h4>
<p>I could definitely be wrong though!</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>